{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to DataFrames\n",
    "**[Bogumił Kamiński](http://bogumilkaminski.pl/about/), November 15, 2020**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling missing values\n",
    "\n",
    "A singelton type `Missing` allows us to deal with missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(missing, Missing)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing, typeof(missing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arrays automatically create an appropriate union type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Union{Missing, Int64},1}:\n",
       " 1\n",
       " 2\n",
       "  missing\n",
       " 3"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [1, 2, missing, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ismissing` checks if passed value is missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(false, true, false, Bool[0, 0, 1, 0])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ismissing(1), ismissing(missing), ismissing(x), ismissing.(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the type combined with Missing from a `Union` via `nonmissingtype`\n",
    "\n",
    "(This is useful for arrays!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Union{Missing, Int64}, Int64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eltype(x), nonmissingtype(eltype(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`missing` comparisons produce `missing`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(missing, missing, missing)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing == missing, missing != missing, missing < missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is also true when `missing`s are compared with values of other types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(missing, missing, missing)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 == missing, 1 != missing, 1 < missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`isequal`, `isless`, and `===` produce results of type `Bool`. Notice that `missing` is considered greater than any numeric value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(true, true, false, true)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isequal(missing, missing), missing === missing, isequal(1, missing), isless(1, missing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next few examples, we see that many (not all) functions handle `missing`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Missing,1}:\n",
       " missing\n",
       " missing\n",
       " missing\n",
       " missing"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(x -> x(missing), [sin, cos, zero, sqrt]) # part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5-element Array{Missing,1}:\n",
       " missing\n",
       " missing\n",
       " missing\n",
       " missing\n",
       " missing"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "map(x -> x(missing, 1), [+, - , *, /, div]) # part 2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5-element Array{Any,1}:\n",
       " missing\n",
       " missing\n",
       " (missing, missing)\n",
       " missing\n",
       " Union{Missing, Float64}[1.0, 2.0, missing]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Statistics # needed for mean\n",
    "map(x -> x([1,2,missing]), [minimum, maximum, extrema, mean, float]) # part 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`skipmissing` returns iterator skipping missing values. We can use `collect` and `skipmissing` to create an array that excludes these missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Array{Int64,1}:\n",
       " 1\n",
       " 2"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collect(skipmissing([1, missing, 2, missing]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use `replace` to create a new array that replaces all missing values with some value (`NaN` in this case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Float64,1}:\n",
       "   1.0\n",
       " NaN\n",
       "   2.0\n",
       " NaN"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace([1.0, missing, 2.0, missing], missing=>NaN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Float64,1}:\n",
       "   1.0\n",
       " NaN\n",
       "   2.0\n",
       " NaN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coalesce.([1.0, missing, 2.0, missing], NaN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `recode` from CategoricalArrays.jl if you have a default output value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "using CategoricalArrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{Bool,1}:\n",
       " 0\n",
       " 1\n",
       " 0\n",
       " 1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recode([1.0, missing, 2.0, missing], false, missing=>true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also `replace!` and `recode!` functions that work in place."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example how you can to missing inputation in a data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Int64?</th><th>String?</th></tr></thead><tbody><p>3 rows × 2 columns</p><tr><th>1</th><td>1</td><td>a</td></tr><tr><th>2</th><td>2</td><td>b</td></tr><tr><th>3</th><td><em>missing</em></td><td><em>missing</em></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & String?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & a \\\\\n",
       "\t2 & 2 & b \\\\\n",
       "\t3 & \\emph{missing} & \\emph{missing} \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a       \u001b[0m\u001b[1m b       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64?  \u001b[0m\u001b[90m String? \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │       1  a\n",
       "   2 │       2  b\n",
       "   3 │\u001b[90m missing \u001b[0m\u001b[90m missing \u001b[0m"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = DataFrame(a=[1,2,missing], b=[\"a\", \"b\", missing])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we change `df.a` vector in place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Union{Missing, Int64},1}:\n",
       "   1\n",
       "   2\n",
       " 100"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace!(df.a, missing=>100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we overwrite `df.b` with a new vector, because the replacement type is different than what `eltype(df.b)` accepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Any,1}:\n",
       "    \"a\"\n",
       "    \"b\"\n",
       " 100"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.b = coalesce.(df.b, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Int64?</th><th>Any</th></tr></thead><tbody><p>3 rows × 2 columns</p><tr><th>1</th><td>1</td><td>a</td></tr><tr><th>2</th><td>2</td><td>b</td></tr><tr><th>3</th><td>100</td><td>100</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & Any\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & a \\\\\n",
       "\t2 & 2 & b \\\\\n",
       "\t3 & 100 & 100 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a      \u001b[0m\u001b[1m b   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64? \u001b[0m\u001b[90m Any \u001b[0m\n",
       "─────┼─────────────\n",
       "   1 │      1  a\n",
       "   2 │      2  b\n",
       "   3 │    100  100"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `unique` or `levels` to get unique values with or without missings, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Union{Missing, Int64}[1, missing, 2], [1, 2])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique([1, missing, 2, missing]), levels([1, missing, 2, missing])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this next example, we convert `x` to `y` with `allowmissing`, where `y` has a type that accepts missings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Union{Missing, Int64},1}:\n",
       " 1\n",
       " 2\n",
       " 3"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = [1,2,3]\n",
    "y = allowmissing(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we convert back with `disallowmissing`. This would fail if `y` contained missing values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([1, 2, 3], Union{Missing, Int64}[1, 2, 3], [1, 2, 3])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = disallowmissing(y)\n",
    "x,y,z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`disallowmissing` has `error` keyword argument that can be used to decide how it should behave when it encounters a column that actually contains a `missing` value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>x1</th><th>x2</th><th>x3</th></tr><tr><th></th><th>Float64?</th><th>Float64?</th><th>Float64?</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><th>2</th><td>1.0</td><td>1.0</td><td>1.0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& x1 & x2 & x3\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64? & Float64?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1.0 & 1.0 & 1.0 \\\\\n",
       "\t2 & 1.0 & 1.0 & 1.0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x1       \u001b[0m\u001b[1m x2       \u001b[0m\u001b[1m x3       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64? \u001b[0m\u001b[90m Float64? \u001b[0m\u001b[90m Float64? \u001b[0m\n",
       "─────┼──────────────────────────────\n",
       "   1 │      1.0       1.0       1.0\n",
       "   2 │      1.0       1.0       1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = allowmissing(DataFrame(ones(2,3), :auto))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "missing"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[1,1] = missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>x1</th><th>x2</th><th>x3</th></tr><tr><th></th><th>Float64?</th><th>Float64?</th><th>Float64?</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td><em>missing</em></td><td>1.0</td><td>1.0</td></tr><tr><th>2</th><td>1.0</td><td>1.0</td><td>1.0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& x1 & x2 & x3\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64? & Float64?\\\\\n",
       "\t\\hline\n",
       "\t1 & \\emph{missing} & 1.0 & 1.0 \\\\\n",
       "\t2 & 1.0 & 1.0 & 1.0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x1        \u001b[0m\u001b[1m x2       \u001b[0m\u001b[1m x3       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64?  \u001b[0m\u001b[90m Float64? \u001b[0m\u001b[90m Float64? \u001b[0m\n",
       "─────┼───────────────────────────────\n",
       "   1 │\u001b[90m missing   \u001b[0m      1.0       1.0\n",
       "   2 │       1.0       1.0       1.0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "MethodError: Cannot `convert` an object of type Missing to an object of type Float64\nClosest candidates are:\n  convert(::Type{T}, !Matched::T) where T<:Number at number.jl:6\n  convert(::Type{T}, !Matched::Number) where T<:Number at number.jl:7\n  convert(::Type{T}, !Matched::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250\n  ...",
     "output_type": "error",
     "traceback": [
      "MethodError: Cannot `convert` an object of type Missing to an object of type Float64\nClosest candidates are:\n  convert(::Type{T}, !Matched::T) where T<:Number at number.jl:6\n  convert(::Type{T}, !Matched::Number) where T<:Number at number.jl:7\n  convert(::Type{T}, !Matched::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250\n  ...",
      "",
      "Stacktrace:",
      " [1] setindex!(::Array{Float64,1}, ::Missing, ::Int64) at .\\array.jl:847",
      " [2] _unsafe_copyto!(::Array{Float64,1}, ::Int64, ::Array{Union{Missing, Float64},1}, ::Int64, ::Int64) at .\\array.jl:257",
      " [3] unsafe_copyto! at .\\array.jl:311 [inlined]",
      " [4] _copyto_impl! at .\\array.jl:335 [inlined]",
      " [5] copyto! at .\\array.jl:321 [inlined]",
      " [6] copyto! at .\\array.jl:347 [inlined]",
      " [7] copyto_axcheck! at .\\abstractarray.jl:946 [inlined]",
      " [8] AbstractArray at .\\array.jl:563 [inlined]",
      " [9] AbstractArray at .\\boot.jl:433 [inlined]",
      " [10] convert at .\\abstractarray.jl:15 [inlined]",
      " [11] disallowmissing(::Array{Union{Missing, Float64},1}) at D:\\AppData\\.julia\\packages\\Missings\\notdc\\src\\Missings.jl:50",
      " [12] disallowmissing(::DataFrame, ::Colon; error::Bool) at D:\\AppData\\.julia\\packages\\DataFrames\\X0xNW\\src\\abstractdataframe\\abstractdataframe.jl:1822",
      " [13] disallowmissing at D:\\AppData\\.julia\\packages\\DataFrames\\X0xNW\\src\\abstractdataframe\\abstractdataframe.jl:1814 [inlined] (repeats 2 times)",
      " [14] top-level scope at In[27]:1",
      " [15] include_string(::Function, ::Module, ::String, ::String) at .\\loading.jl:1091",
      " [16] execute_code(::String, ::String) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\execute_request.jl:27",
      " [17] execute_request(::ZMQ.Socket, ::IJulia.Msg) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\execute_request.jl:86",
      " [18] #invokelatest#1 at .\\essentials.jl:710 [inlined]",
      " [19] invokelatest at .\\essentials.jl:709 [inlined]",
      " [20] eventloop(::ZMQ.Socket) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\eventloop.jl:8",
      " [21] (::IJulia.var\"#15#18\")() at .\\task.jl:356"
     ]
    }
   ],
   "source": [
    "disallowmissing(df) # an error is thrown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>x1</th><th>x2</th><th>x3</th></tr><tr><th></th><th>Float64?</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td><em>missing</em></td><td>1.0</td><td>1.0</td></tr><tr><th>2</th><td>1.0</td><td>1.0</td><td>1.0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& x1 & x2 & x3\\\\\n",
       "\t\\hline\n",
       "\t& Float64? & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & \\emph{missing} & 1.0 & 1.0 \\\\\n",
       "\t2 & 1.0 & 1.0 & 1.0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x1        \u001b[0m\u001b[1m x2      \u001b[0m\u001b[1m x3      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64?  \u001b[0m\u001b[90m Float64 \u001b[0m\u001b[90m Float64 \u001b[0m\n",
       "─────┼─────────────────────────────\n",
       "   1 │\u001b[90m missing   \u001b[0m     1.0      1.0\n",
       "   2 │       1.0      1.0      1.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disallowmissing(df, error=false) # column :x1 is left untouched as it contains missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this next example, we show that the type of each column in `x` is initially `Int64`. After using `allowmissing!` to accept missing values in columns 1 and 3, the types of those columns become `Union{Int64,Missing}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: DataType[Int64, Int64, Int64]\n",
      "After: Type[Union{Missing, Int64}, Int64, Union{Missing, Int64}]\n"
     ]
    }
   ],
   "source": [
    "x = DataFrame(rand(Int, 2,3), :auto)\n",
    "println(\"Before: \", eltype.(eachcol(x)))\n",
    "allowmissing!(x, 1) # make first column accept missings\n",
    "allowmissing!(x, :x3) # make :x3 column accept missings\n",
    "println(\"After: \", eltype.(eachcol(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this next example, we'll use `completecases` to find all the rows of a `DataFrame` that have complete data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64?</th><th>String?</th></tr></thead><tbody><p>4 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td><em>missing</em></td><td>B</td></tr><tr><th>3</th><td>3</td><td><em>missing</em></td></tr><tr><th>4</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & String?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & \\emph{missing} & B \\\\\n",
       "\t3 & 3 & \\emph{missing} \\\\\n",
       "\t4 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A       \u001b[0m\u001b[1m B       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64?  \u001b[0m\u001b[90m String? \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │       1  A\n",
       "   2 │\u001b[90m missing \u001b[0m B\n",
       "   3 │       3 \u001b[90m missing \u001b[0m\n",
       "   4 │       4  C"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = DataFrame(A=[1, missing, 3, 4], B=[\"A\", \"B\", missing, \"C\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Complete cases:\n",
      "Bool[1, 0, 0, 1]\n"
     ]
    }
   ],
   "source": [
    "println(\"Complete cases:\\n\", completecases(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `dropmissing` or `dropmissing!` to remove the rows with incomplete data from a `DataFrame` and either create a new `DataFrame` or mutate the original in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A     \u001b[0m\u001b[1m B      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String \u001b[0m\n",
       "─────┼───────────────\n",
       "   1 │     1  A\n",
       "   2 │     4  C"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = dropmissing(x)\n",
    "dropmissing!(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A     \u001b[0m\u001b[1m B      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String \u001b[0m\n",
       "─────┼───────────────\n",
       "   1 │     1  A\n",
       "   2 │     4  C"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A     \u001b[0m\u001b[1m B      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String \u001b[0m\n",
       "─────┼───────────────\n",
       "   1 │     1  A\n",
       "   2 │     4  C"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we call `describe` on a `DataFrame` with dropped missing values, the columns do not allow missing values any more by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>variable</th><th>mean</th><th>min</th><th>median</th><th>max</th><th>nmissing</th><th>eltype</th></tr><tr><th></th><th>Symbol</th><th>Union…</th><th>Any</th><th>Union…</th><th>Any</th><th>Int64</th><th>DataType</th></tr></thead><tbody><p>2 rows × 7 columns</p><tr><th>1</th><td>A</td><td>2.5</td><td>1</td><td>2.5</td><td>4</td><td>0</td><td>Int64</td></tr><tr><th>2</th><td>B</td><td></td><td>A</td><td></td><td>C</td><td>0</td><td>String</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& variable & mean & min & median & max & nmissing & eltype\\\\\n",
       "\t\\hline\n",
       "\t& Symbol & Union… & Any & Union… & Any & Int64 & DataType\\\\\n",
       "\t\\hline\n",
       "\t1 & A & 2.5 & 1 & 2.5 & 4 & 0 & Int64 \\\\\n",
       "\t2 & B &  & A &  & C & 0 & String \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×7 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean   \u001b[0m\u001b[1m min \u001b[0m\u001b[1m median \u001b[0m\u001b[1m max \u001b[0m\u001b[1m nmissing \u001b[0m\u001b[1m eltype   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Symbol   \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Any \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Any \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m DataType \u001b[0m\n",
       "─────┼────────────────────────────────────────────────────────\n",
       "   1 │ A         2.5     1    2.5     4           0  Int64\n",
       "   2 │ B        \u001b[90m        \u001b[0m A   \u001b[90m        \u001b[0m C           0  String"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "describe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively you can pass `disallowmissing` keyword argument to `dropmissing` and `dropmissing!`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64?</th><th>String?</th></tr></thead><tbody><p>4 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td><em>missing</em></td><td>B</td></tr><tr><th>3</th><td>3</td><td><em>missing</em></td></tr><tr><th>4</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & String?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & \\emph{missing} & B \\\\\n",
       "\t3 & 3 & \\emph{missing} \\\\\n",
       "\t4 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A       \u001b[0m\u001b[1m B       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64?  \u001b[0m\u001b[90m String? \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │       1  A\n",
       "   2 │\u001b[90m missing \u001b[0m B\n",
       "   3 │       3 \u001b[90m missing \u001b[0m\n",
       "   4 │       4  C"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = DataFrame(A=[1, missing, 3, 4], B=[\"A\", \"B\", missing, \"C\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>A</th><th>B</th></tr><tr><th></th><th>Int64?</th><th>String?</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>1</td><td>A</td></tr><tr><th>2</th><td>4</td><td>C</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& A & B\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & String?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & A \\\\\n",
       "\t2 & 4 & C \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m A      \u001b[0m\u001b[1m B       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64? \u001b[0m\u001b[90m String? \u001b[0m\n",
       "─────┼─────────────────\n",
       "   1 │      1  A\n",
       "   2 │      4  C"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropmissing!(x, disallowmissing=false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making functions `missing`-aware\n",
    "\n",
    "If we have a function that does not handle `missing` values we can wrap it using `passmissing` function so that if any of its positional arguments is missing we will get a `missing` value in return. In the example below we change how `string` function behaves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"missing\""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string(missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"missing missing\""
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string(missing, \" \", missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"123\""
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string(1,2,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(::Missings.PassMissing{typeof(string)}) (generic function with 2 methods)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lift_string = passmissing(string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "missing"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lift_string(missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "missing"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lift_string(missing, \" \", missing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"123\""
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lift_string(1,2,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregating rows containing missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an example data frame containing missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Int64?</th><th>Int64?</th></tr></thead><tbody><p>3 rows × 2 columns</p><tr><th>1</th><td>1</td><td>1</td></tr><tr><th>2</th><td><em>missing</em></td><td>2</td></tr><tr><th>3</th><td><em>missing</em></td><td><em>missing</em></td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Int64? & Int64?\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1 \\\\\n",
       "\t2 & \\emph{missing} & 2 \\\\\n",
       "\t3 & \\emph{missing} & \\emph{missing} \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a       \u001b[0m\u001b[1m b       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64?  \u001b[0m\u001b[90m Int64?  \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │       1        1\n",
       "   2 │\u001b[90m missing \u001b[0m       2\n",
       "   3 │\u001b[90m missing \u001b[0m\u001b[90m missing \u001b[0m"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = DataFrame(a = [1,missing,missing], b=[1,2,missing])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we just run `sum` on the rows we get two missing entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Union{Missing, Int64},1}:\n",
       " 2\n",
       "  missing\n",
       "  missing"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum.(eachrow(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can apply `skipmissing` on the rows to avoid this problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "ArgumentError: reducing over an empty collection is not allowed",
     "output_type": "error",
     "traceback": [
      "ArgumentError: reducing over an empty collection is not allowed",
      "",
      "Stacktrace:",
      " [1] _empty_reduce_error() at .\\reduce.jl:299",
      " [2] reduce_empty_iter(::Base.BottomRF{typeof(Base.add_sum)}, ::Base.SkipMissing{DataFrameRow{DataFrame,DataFrames.Index}}, ::Base.EltypeUnknown) at .\\reduce.jl:356",
      " [3] reduce_empty_iter at .\\reduce.jl:354 [inlined]",
      " [4] foldl_impl at .\\reduce.jl:49 [inlined]",
      " [5] mapfoldl_impl at .\\reduce.jl:44 [inlined]",
      " [6] #mapfoldl#204 at .\\reduce.jl:160 [inlined]",
      " [7] mapfoldl at .\\reduce.jl:160 [inlined]",
      " [8] #mapreduce#208 at .\\reduce.jl:287 [inlined]",
      " [9] mapreduce at .\\reduce.jl:287 [inlined]",
      " [10] sum at .\\reduce.jl:494 [inlined]",
      " [11] sum at .\\reduce.jl:511 [inlined]",
      " [12] _broadcast_getindex_evalf at .\\broadcast.jl:648 [inlined]",
      " [13] _broadcast_getindex at .\\broadcast.jl:621 [inlined]",
      " [14] getindex at .\\broadcast.jl:575 [inlined]",
      " [15] copyto_nonleaf!(::Array{Int64,1}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(sum),Tuple{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(skipmissing),Tuple{Base.Broadcast.Extruded{DataFrames.DataFrameRows{DataFrame,DataFrames.Index},Tuple{Bool},Tuple{Int64}}}}}}, ::Base.OneTo{Int64}, ::Int64, ::Int64) at .\\broadcast.jl:1026",
      " [16] copy at .\\broadcast.jl:880 [inlined]",
      " [17] materialize(::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(sum),Tuple{Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(skipmissing),Tuple{DataFrames.DataFrameRows{DataFrame,DataFrames.Index}}}}}) at .\\broadcast.jl:837",
      " [18] top-level scope at In[47]:1",
      " [19] include_string(::Function, ::Module, ::String, ::String) at .\\loading.jl:1091",
      " [20] execute_code(::String, ::String) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\execute_request.jl:27",
      " [21] execute_request(::ZMQ.Socket, ::IJulia.Msg) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\execute_request.jl:86",
      " [22] #invokelatest#1 at .\\essentials.jl:710 [inlined]",
      " [23] invokelatest at .\\essentials.jl:709 [inlined]",
      " [24] eventloop(::ZMQ.Socket) at D:\\AppData\\.julia\\packages\\IJulia\\a1SNk\\src\\eventloop.jl:8",
      " [25] (::IJulia.var\"#15#18\")() at .\\task.jl:356"
     ]
    }
   ],
   "source": [
    "sum.(skipmissing.(eachrow(df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we get an error. The problem is that the last row of `df` contains only missing values, and since `eachrow` is type unstable the `eltype` of the result of `skipmissing` is unknown (so it is marked `Any`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Any[]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collect(skipmissing(eachrow(df)[end]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In such cases it is useful to switch to `Tables.namedtupleiterator` which is type stable as discussed in 01_constructors.ipynb notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Int64,1}:\n",
       " 2\n",
       " 2\n",
       " 0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum.(skipmissing.(Tables.namedtupleiterator(df)))"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.5",
   "language": "julia",
   "name": "julia-1.5"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
